Ensemble Control of Cycling Energy Loads: Markov Decision Approach

Markov Decision Process (MDP) framework is adopted to represent ensemble control of devices, whose energy consumption pattern is of a cycling type, e.g. thermostatically controlled loads. Specifically we utilize and develop the class of MDP models, coined previously linearly-solvable MDP, that describe optimal dynamics of the probability distribution of an ensemble of many cycling devices. Two principally different settings are discussed. First, we consider optimal strategy of the ensemble aggregator aimed at balancing between minimizing the common objective that consists of the cost of the ensemble consumption, under varying price of electricity, and the welfare deviation from normal, evaluated as the KL-divergence between the optimal and nominal transition probabilities between different states of the ensemble’s participants. Second, we shift to the Demand Response setting, where the aggregator aims to minimize the welfare deviation under the condition that the aggregated consumption of the ensemble matches the perfectly varying in time target consumption, requested by the system operator. We discuss a modification of both settings, aimed at encouraging or constraining the transitions between different states. The dynamic programming feature of the resulting modified MDPs is always preserved, however lin(ear)-solvability is lost fully or partially, depending on the type of modification. We also conducted some (limited in scope) numerical experimentation using the formulations of the first setting. We conclude with discussing future generalizations and applications. Michael Chertkov Theoretical Division, T-4 & CNLS, Los Alamos National Laboratory Los Alamos, NM 87545, USA and Energy System Center, Skoltech, Moscow, 143026, Russia, e-mail: chertkov@lanl.